这项工作提出了专门针对粒子探测器的低潜伏期图神经网络（GNN）设计的新型可重构体系结构。加速粒子探测器的GNN是具有挑战性的，因为它需要次微秒延迟才能在CERN大型强子撞机实验的级别1触发器中部署网络以进行在线事件选择。本文提出了一种自定义代码转换，并在基于互动网络的GNN中使用完全连接的图表中的矩阵乘法操作降低了强度，从而避免了昂贵的乘法。它利用了稀疏模式以及二进制邻接矩阵，并避免了不规则的内存访问，从而降低了延迟和硬件效率的提高。此外，我们引入了一种基于外部产品的基质乘法方法，该方法通过降低潜伏期设计的强度降低来增强。此外，引入了融合步骤，以进一步降低设计延迟。此外，提出了GNN特异性算法 - 硬件共同设计方法，该方法不仅找到了具有更好延迟的设计，而且在给定的延迟约束下发现了高精度的设计。最后，已经设计和开源了此低延迟GNN硬件体系结构的可自定义模板，该模板可以使用高级合成工具来生成低延迟的FPGA设计，并有效地利用资源。评估结果表明，我们的FPGA实施速度高24倍，并且消耗的功率比GPU实施少45倍。与我们以前的FPGA实施相比，这项工作的延迟降低了6.51至16.7倍。此外，我们的FPGA设计的延迟足以使GNN在亚微秒，实时撞机触发器系统中部署，从而使其能够从提高的精度中受益。

深度神经网络（DNN）的算法 - 硬件共同设计的最新进展已经证明了它们在自动设计神经架构和硬件设计方面的潜力。然而，由于昂贵的培训成本和耗时的硬件实现，这仍然是一个充满挑战的优化问题，这使得对神经结构和硬件设计难以解答的巨大设计空间探索。在本文中，我们证明我们所提出的方法能够在帕累托前沿定位设计。这种功能由新颖的三相协同设计框架启用，具有以下新功能：（a）从硬件架构和神经结构的设计空间探索的DNN培训解耦，（b）提供硬件友好的神经结构空间通过考虑构造搜索单元的硬件特征，（c）采用高斯过程来预测准确性，延迟和功耗以避免耗时的合成和路由过程。与手动设计的Resnet101，Inceptionv2和MobileNetv2相比，我们可以在想象网数据集中获得高达3倍的准确度，高达5％的准确性。与其他最先进的共同设计框架相比，我们发现的网络和硬件配置可以达到更高的2％〜6％，精度为2倍〜26倍，延迟较高8.5倍。

神经网络在广泛的任务中展示了他们出色的表现。具体地，基于长短短期存储器（LSTM）单元格的复发架构表现出了在真实数据中模拟时间依赖性的优异能力。然而，标准的经常性架构无法估计其不确定性，这对于安全关键型应用如医学，这是必不可少的。相比之下，贝叶斯经常性神经网络（RNN）能够以提高的精度提供不确定性估计。尽管如此，贝叶斯的RNN是在计算上和记忆所要求的，尽管他们的优势尽管他们的实用性限制了他们的实用性。为了解决这个问题，我们提出了一种基于FPGA的硬件设计，以加速基于贝叶斯LSTM的RNN。为了进一步提高整体算法 - 硬件性能，提出了一种共同设计框架来探索贝叶斯RNN的最适合的算法 - 硬件配置。我们对医疗保健应用进行了广泛的实验，以证明我们的设计和框架的有效性的提高。与GPU实施相比，我们的FPGA的设计可以实现高达10倍的加速，能效率较高的近106倍。据我们所知，这是第一份针对FPGA上的贝叶斯RNN的加速的工作。

The ability to jointly learn from multiple modalities, such as text, audio, and visual data, is a defining feature of intelligent systems. While there have been promising advances in designing neural networks to harness multimodal data, the enormous success of data augmentation currently remains limited to single-modality tasks like image classification. Indeed, it is particularly difficult to augment each modality while preserving the overall semantic structure of the data; for example, a caption may no longer be a good description of an image after standard augmentations have been applied, such as translation. Moreover, it is challenging to specify reasonable transformations that are not tailored to a particular modality. In this paper, we introduce LeMDA, Learning Multimodal Data Augmentation, an easy-to-use method that automatically learns to jointly augment multimodal data in feature space, with no constraints on the identities of the modalities or the relationship between modalities. We show that LeMDA can (1) profoundly improve the performance of multimodal deep learning architectures, (2) apply to combinations of modalities that have not been previously considered, and (3) achieve state-of-the-art results on a wide range of applications comprised of image, text, and tabular data.

The number of international benchmarking competitions is steadily increasing in various fields of machine learning (ML) research and practice. So far, however, little is known about the common practice as well as bottlenecks faced by the community in tackling the research questions posed. To shed light on the status quo of algorithm development in the specific field of biomedical imaging analysis, we designed an international survey that was issued to all participants of challenges conducted in conjunction with the IEEE ISBI 2021 and MICCAI 2021 conferences (80 competitions in total). The survey covered participants' expertise and working environments, their chosen strategies, as well as algorithm characteristics. A median of 72% challenge participants took part in the survey. According to our results, knowledge exchange was the primary incentive (70%) for participation, while the reception of prize money played only a minor role (16%). While a median of 80 working hours was spent on method development, a large portion of participants stated that they did not have enough time for method development (32%). 25% perceived the infrastructure to be a bottleneck. Overall, 94% of all solutions were deep learning-based. Of these, 84% were based on standard architectures. 43% of the respondents reported that the data samples (e.g., images) were too large to be processed at once. This was most commonly addressed by patch-based training (69%), downsampling (37%), and solving 3D analysis tasks as a series of 2D tasks. K-fold cross-validation on the training set was performed by only 37% of the participants and only 50% of the participants performed ensembling based on multiple identical models (61%) or heterogeneous models (39%). 48% of the respondents applied postprocessing steps.

Multimodal image-text models have shown remarkable performance in the past few years. However, evaluating their robustness against distribution shifts is crucial before adopting them in real-world applications. In this paper, we investigate the robustness of 9 popular open-sourced image-text models under common perturbations on five tasks (image-text retrieval, visual reasoning, visual entailment, image captioning, and text-to-image generation). In particular, we propose several new multimodal robustness benchmarks by applying 17 image perturbation and 16 text perturbation techniques on top of existing datasets. We observe that multimodal models are not robust to image and text perturbations, especially to image perturbations. Among the tested perturbation methods, character-level perturbations constitute the most severe distribution shift for text, and zoom blur is the most severe shift for image data. We also introduce two new robustness metrics (MMI and MOR) for proper evaluations of multimodal models. We hope our extensive study sheds light on new directions for the development of robust multimodal models.

Real-time monocular 3D reconstruction is a challenging problem that remains unsolved. Although recent end-to-end methods have demonstrated promising results, tiny structures and geometric boundaries are hardly captured due to their insufficient supervision neglecting spatial details and oversimplified feature fusion ignoring temporal cues. To address the problems, we propose an end-to-end 3D reconstruction network SST, which utilizes Sparse estimated points from visual SLAM system as additional Spatial guidance and fuses Temporal features via a novel cross-modal attention mechanism, achieving more detailed reconstruction results. We propose a Local Spatial-Temporal Fusion module to exploit more informative spatial-temporal cues from multi-view color information and sparse priors, as well a Global Spatial-Temporal Fusion module to refine the local TSDF volumes with the world-frame model from coarse to fine. Extensive experiments on ScanNet and 7-Scenes demonstrate that SST outperforms all state-of-the-art competitors, whilst keeping a high inference speed at 59 FPS, enabling real-world applications with real-time requirements.

Image-text retrieval in remote sensing aims to provide flexible information for data analysis and application. In recent years, state-of-the-art methods are dedicated to ``scale decoupling'' and ``semantic decoupling'' strategies to further enhance the capability of representation. However, these previous approaches focus on either the disentangling scale or semantics but ignore merging these two ideas in a union model, which extremely limits the performance of cross-modal retrieval models. To address these issues, we propose a novel Scale-Semantic Joint Decoupling Network (SSJDN) for remote sensing image-text retrieval. Specifically, we design the Bidirectional Scale Decoupling (BSD) module, which exploits Salience Feature Extraction (SFE) and Salience-Guided Suppression (SGS) units to adaptively extract potential features and suppress cumbersome features at other scales in a bidirectional pattern to yield different scale clues. Besides, we design the Label-supervised Semantic Decoupling (LSD) module by leveraging the category semantic labels as prior knowledge to supervise images and texts probing significant semantic-related information. Finally, we design a Semantic-guided Triple Loss (STL), which adaptively generates a constant to adjust the loss function to improve the probability of matching the same semantic image and text and shorten the convergence time of the retrieval model. Our proposed SSJDN outperforms state-of-the-art approaches in numerical experiments conducted on four benchmark remote sensing datasets.

Although existing semi-supervised learning models achieve remarkable success in learning with unannotated in-distribution data, they mostly fail to learn on unlabeled data sampled from novel semantic classes due to their closed-set assumption. In this work, we target a pragmatic but under-explored Generalized Novel Category Discovery (GNCD) setting. The GNCD setting aims to categorize unlabeled training data coming from known and novel classes by leveraging the information of partially labeled known classes. We propose a two-stage Contrastive Affinity Learning method with auxiliary visual Prompts, dubbed PromptCAL, to address this challenging problem. Our approach discovers reliable pairwise sample affinities to learn better semantic clustering of both known and novel classes for the class token and visual prompts. First, we propose a discriminative prompt regularization loss to reinforce semantic discriminativeness of prompt-adapted pre-trained vision transformer for refined affinity relationships. Besides, we propose a contrastive affinity learning stage to calibrate semantic representations based on our iterative semi-supervised affinity graph generation method for semantically-enhanced prompt supervision. Extensive experimental evaluation demonstrates that our PromptCAL method is more effective in discovering novel classes even with limited annotations and surpasses the current state-of-the-art on generic and fine-grained benchmarks (with nearly $11\%$ gain on CUB-200, and $9\%$ on ImageNet-100) on overall accuracy.

Multi-view representation learning has developed rapidly over the past decades and has been applied in many fields. However, most previous works assumed that each view is complete and aligned. This leads to an inevitable deterioration in their performance when encountering practical problems such as missing or unaligned views. To address the challenge of representation learning on partially aligned multi-view data, we propose a new cross-view graph contrastive learning framework, which integrates multi-view information to align data and learn latent representations. Compared with current approaches, the proposed method has the following merits: (1) our model is an end-to-end framework that simultaneously performs view-specific representation learning via view-specific autoencoders and cluster-level data aligning by combining multi-view information with the cross-view graph contrastive learning; (2) it is easy to apply our model to explore information from three or more modalities/sources as the cross-view graph contrastive learning is devised. Extensive experiments conducted on several real datasets demonstrate the effectiveness of the proposed method on the clustering and classification tasks.